Operating in the real-world often requires agents to learn about a complex environment and apply this understanding to achieve a breadth of goals.

The generalization ability of the agent is measured by making the set of backgrounds in the R2 seen task and the set of backgrounds in the R2 unseen task mutually exclusive.

We agree that it is a nice idea to exploit the bounding boxes of ImageNet, and are happy to explore it in GFNet. We will add more comparisons on this point in our revision. The iterative process is indeed not indispensable, but in experiments, it improves the accuracy (e.g., We will make these clear in revision. We will release all the code and pre-trained models upon the acceptance of this paper. Table 1: Results using 32x32 (left) and 64x64 (right) patches.

Our innovation of optimizing interval times is highlighted in blue in Algorithm 1 . A key assumption of Algorithm 1 is that acting often using short time intervals will not hurt performance, and that maximal interaction (i.e. In many scenarios, this assumption seems reasonable and applying Algorithm 1 may work well. For example, some Atari games require frameskipping, i.e., repeating actions for Algorithm 2 assumes that the dynamics can be fully covered by random policies. However, these may be far away from the optimal policy.

Pretrained foundation models (FMs) have exhibited extraordinary in-context learning performance, allowing zero-shot (or few-shot) generalization to new environments/tasks not encountered during the pretraining. In the case of reinforcement learning (RL), in-context RL (ICRL) emerges when pretraining FMs on decision-making problems in an autoregressivesupervised manner. Nevertheless, the current state-of-the-art ICRL algorithms, such as Algorithm Distillation, Decision Pretrained Transformer and Decision Importance Transformer, impose stringent requirements on the pretraining dataset concerning the behavior (source) policies, context information, and action labels, etc. Notably, these algorithms either demand optimal policies or require varying degrees of well-trained behavior policies for all pretraining environments. This significantly hinders the application of ICRL to realworld scenarios, where acquiring optimal or well-trained policies for a substantial volume of real-world training environments can be prohibitively expensive or even intractable. To overcome this challenge, we introduce a novel approach, termed State-Action Distillation (SAD), that allows to generate an effective pretraining dataset guided solely by random policies. In particular, SAD selects query states and corresponding action labels by distilling the outstanding state-action pairs from the entire state and action spaces by using random policies within a trust horizon, and then inherits the classical autoregressive-supervised mechanism during the pretraining. To the best of our knowledge, this is the first work that enables effective ICRL under (e.g., uniform) random policies and random contexts. We also establish the quantitative analysis of the trustworthiness as well as the performance guarantees of our SAD approach. Moreover, our empirical results across multiple popular ICRL benchmark environments demonstrate that, on average, SAD outperforms the best baseline by 236.3% in the offline evaluation and by 135.2% in the online evaluation.